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ABSTRACT 



This report describes the implementation of the DC 21st 
Century Learning Centers (DC 21st CCLC) Summer 2000 program, which enrolled 
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is designed to provide feedback to managers of the DC 21st CCLC program and 
to inform Children and Youth Investment Partnership, of which DC 21st CCLC is 
a part. The DC 21st CCLC program, run by the District of Columbia public 
school system (DCPS) , began as an initiative to help coordinate the different 
types of youth activities provided for children in the District of Columbia 
and currently includes after-school, summer, Saturday, and adult evening 
programs. The report, based on observations of program activities and 
interviews with coordinators, facilitators, and parents at the 9 DC 21st CCLC 
summer schools, concludes that in general the program has been well 
implemented with sufficient resources and support, and that it has responded 
with flexibility to challenges beyond the program's control. Parent responses 
were positive, staff appeared to be satisfied with program implementation, 
and activities appeared to give students many opportunities to develop a 
variety of skills in the areas of education, technology, sports and community 
service. Appendixes present characteristics of observed program activities, a 
description of technology use in DC 21st CCLC program, and protocols for site 
visits. (SM) 
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Abstract 



This report describes the implementation of the DC 21 st Century Community Learning 
Center (DC 21 st CCLC) Summer 2000 Program. The report is designed to provide 
feedback to the managers of the DC 21 st CCLC program, and to inform Children and 
Youth Investment Partnership activities, of which the DC 21 st CCLC program is a part. 
The report is based on observations of program activities and interviews with program 
coordinators, facilitators, and parents at the nine DC 21 st CCLC summer 2000 school 
programs. The results suggest that in general the program has been implemented with 
sufficient resources and support, and that it has responded with flexibility to 
implementation challenges beyond the program’s control. Parent reports were positive, 
staff appeared satisfied with program implementation, and activities appeared to give 
students many opportunities to develop a variety of skills. The next step in an evaluation 
is to explore the use of a comparison group to document results more fully. 
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Formative Report on the District of Columbia 
21 st Century Community Learning Center Summer Program 

The Urban Institute 



The purpose of this report is to summarize key issues that emerged from The Urban 
Institute’s visits to the five sites of the District of Columbia’s 21 st Century Community 
Learning Center program (DC 21 st CCLC) in July 2000. These sites served students, ages 
4 through 13, from nine junior high/middle schools and their feeder schools. 1 

During the site visits, information was gathered to answer the following questions: 

Organizational Issues 

How satisfied were school staff (those implementing the program at the schools) with 
the support provided to them by the District of Columbia Public School (DCPS) staff 
who operate the program, 2 including training, facilities, equipment, etc.? What 
suggestions were made by school staff to improve the assistance and support provided 
by DCPS? 

Potential Student Outcomes 

What types and quality of activities were provided to students in the summer 
program? What potential student outcomes were observed by researchers and reported 
by parents and/or guardians and could be explored in a summative evaluation of the 
program? 

During site visits, Urban Institute researchers undertook the following activities: 

■ Review of school site proposals and DCPS documents detailing professional 
development activities for staff. 

■ Short interviews with nine “Assistant Principals” (“APs,” or program managers) 
and twelve “facilitators” (instructors) from all but one school program. 

■ Observation of 30- to 60-minute segments of 22 program activities, including at 
least two activities at each school program. 

■ Interviews with DCPS staff who operate the DC 21 st CCLC program. 



In April-May, 2000, Urban Institute researchers conducted site visits to these same nine junior 
high/middle schools when they were implementing the after-school component of DC 21 st CCLC. 

2 The Director of the DC 21 st CCLC is Howard M. Brown. The Program Manager is Saundra Handy. 
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I. Background of the DC 21 st CCLC Program 



T he DC 21 st Century Learning Centers (DC 21 st CCLC) program began in 1999 
as part of a broader initiative called the DC Children and Youth Investment 
Partnership (DC CYIP), which is designed to help coordinate the many 
different types of youth activities provided for children during non-school hours in the 
District of Columbia. A brief discussion of DC CYIP and earlier initiatives is provided in 
our formative report on the after-school component of the DC 21 st CCLC program 
(Raphael & Chaplin, 2000). 

The DC 21 st CCLC program is run by the District of Columbia Public School System 
(DCPS) and includes after-school, summer, Saturday, and adult evening sub-programs. 
The after-school program was established in 10 middle/junior high schools with high 
need for such services in November 1999. 3 This report focuses on the summer sub- 
program, which ran in 9 of the 10 original schools, from June 26 to August 4, 2000. 

In addition to being a subset of the DC CYIP initiatives, the DC 21 st CCLC is a subset of 
the out-of-school-time programs operated by DCPS on behalf of the District of Columbia 
Department of Human Services (DHS), which were implemented at forty-seven (47) 
other DCPS sites in addition to the DC 21 st CCLC school sites. Each of these programs is 
funded through Temporary Assistance for Needy Families (TANF) funds. The summer 
operation is collectively referred to as the Summer Education/ Arts/Sports (SEAS) 
program, which started in 1998 with 30 schools. 



II. Program Structure 




ach DC 21 st CCLC school’s summer program was managed by an Assistant 
Principal (AP), who supervised a staff of instructors (facilitators). The AP was 
responsible for recruiting facilitators and students and for organizing special 
events. Most APs hired teachers from their schools to work as facilitators. 4 



The DC 21 st CCLC program is implemented through a collaborative program planning 
process that is overseen by DCPS staff. Through workshops and written materials, APs 
receive guidance on how to structure their programs, particularly regarding the types of, 
activities to be offered. These included activities in: 



• Technofluency (typing, E-mail, and Internet skills); 

• Sports/health (physical and mental well-being); 

• Arts (visual, performing, and other types); and 

• Community service. 



3 Need was determined based on factors such as poverty and test scores of students at these schools. 

4 

In addition, the APs were invited by DCPS to meet and hire artists for their summer programs from a pre- 
selected roster during a specially arranged meeting at Gallaudet University in May of 2000. 
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During spring 2000, each AP wrote a proposal (including a budget) for his or her school 
site’s summer program. DCPS 21 st CCLC administrative staff reviewed these proposals, 
with assistance from outsiders with relevant experience, and made suggestions for 
revisions that led to final approval of the plans and the implementation of the program. 

(A similar procedure was used for the after-school component of DC 21 st CCLC.) 

The DCPS program provided training to APs and reviewed program implementation in 
monthly meetings and special workshops. During the four weeks of this summer 
program, APs attended nightly meetings from 7 to 9 p.m. to review the summer program 5 
and to prepare for the adult component of the program, scheduled to start in the fall of 
2000. In addition, DCPS staff indicated that they encourage APs to contact them at any 
time to discuss program implementation issues. 

DCPS staff also provided training for facilitators. Some of the athletics facilitators, for 
example, reported that they attended a pre-summer program orientation session with 
Frank Briscoe, who organized all summer athletic programs being run and operated by 
DC schools. According to Mr. Briscoe, this training covered the philosophy of the 
program, different activities to be conducted, and assistance to be provided to sports 
coordinators. In addition, DCPS staff reported that they provided four days of training to 
the facilitators who had been identified by that time during the week of June 1 1 (before 
the program began). 



III. Organizational Issues 

T his section identifies key implementation issues that may be of interest to 
DCPS staff for future program implementation. These issues include the 
capacity of the program to serve the target population, school/community 
issues, program planning, staffing, and the extent and quality of support provided to 
school staff. 

Several of the issues discussed in this section may represent opportunities for program 
improvement (e.g., better communication between DCPS and program staff). Providing 
such feedback is an important purpose of this formative report. This discussion does not 
imply , however , that the program has suffered major implementation problems . In fact, 
we found that program staff have responded with flexibility to common implementation 
challenges, and that the program has been implemented generally as planned in nine of 
the ten intended sites. 6 



5 We were told by DCPS that during these sessions APs were provided with information about the 
technology and sports component of the program, how to document their activities, how to manage and train 
their staff, and related information. 

6 MacFarland did not have a program for several reasons: the original coordinator resigned, the school was 
slated to be closed for more than a year, and another summer program for kids (the STARS program 
described below) for MacFarland was located at Lincoln, which is far from the MacFarland neighborhood. 
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Capacity to Serve the Target Audience 

While the DC 21 st CCLC program has received a great deal of support from DCPS, some 
implementation issues arose that influenced the program’s ability to serve its target 
population. These include: the grades served, the site locations, and the location of 
complementary programs. 

Grades Served 

The original plan of the DC 21 st CCLC program was to serve 2,000 youth in grades 6 
through 8 during the school year and 2,000 during the summer (DCPS, 1999). However, 
DCPS staff revised their goals for the summer program in two ways. First, in the 
proposals for the summer programs, they reduced the total goal to between 1,000 and 
1,400 students. Second, they expanded the program to include students from all 
elementary schools that feed into the original middle and junior high schools. 7 They have 
come quite close to meeting the revised goal, serving about 1,000 students regularly. 



Table 1 — Estimates of Students Served 

School Attendance Enrollment Goal 



Eliot 


253 


200 


Francis + Shaw 


150 


100-200 


Garnet-Patterson 


109 


100-200 


P.R. Harris 


211 


200 


Hart 


72 


200 


Kramer + Sousa 


101 


100-200 


Terrell 


93 


100-200 


Total 


989 


1 ,000-1 ,400 



Including students from any elementary school allowed DC 21 st CCLC to serve more 
students than it most likely would have served otherwise, given projections based on 
enrollment in the after-school program. 8 It also changed the nature of the program 
somewhat without detracting from achievement of its goals and objectives. Most of the 
time, students engaged in program activities with similar-age peers, but they also 
interacted with students of varying ages during field trips, athletic competitions, and 
certain other activities. Although serving a wide age range posed challenges for some 
facilitators (most of whom were teachers at junior high/middle schools), when asked, 
most facilitators reported that they enjoyed working with a range of student ages and were 
successful with them. Some parents, too, said that they appreciated the efficiency and 
potential familial benefits of having their various children in the same program together. 

In one case, however, facilitators may have been confused about the target population. 
Four facilitators — three at one site appear to have been given the initial impression, 
perhaps by their AP, that they would be able to choose the age groups with which they 
would work. We did not receive any reports that this had actually occurred at any 



7 This decision was made by the DCPS staff running the DC 21 st CCLC program and is reflected in the 
individual school proposals for the summer programs. 

8 The program fell somewhat short of its school-year attendance goal (Raphael & Chaplin, 2000). 
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program site. The three facilitators at this particular site indicated that they were less 
comfortable serving younger students. The technology facilitator, for example, said it was 
more difficult for him to engage these students, particularly because this site had longer 
activity periods than others. 

Site Location 

The second factor influencing the program’s capacity to serve targeted students is site 
location. In the initial design of the summer component, each of the junior high/middle 
school sites was supposed to implement its own summer program. However, in January 
2000, the Superintendent of DCPS alerted the district that 10 percent of its schools would 
not operate programs in the summer to facilitate needed repairs. DCPS staff operating the 
DC 21 st CCLC program learned in March that three of its schools would be closed. 9 In 
response, DCPS staff combined four school programs and operated them out of one 
school site, and combined two other school programs at another site. In total, nine school 
programs were operated at five school sites. 

The “relocation” of some school programs to sites other than the “home” school meant 
that some students had to travel somewhat significant distances to attend the program. 
Several APs indicated that the relocation limited student enrollment, as parents were 
either unable to transport their children to the alternative site or unwilling to let their 
children (particularly younger children) use public transportation to get to the site. 

Complementary Program Location 

DCPS also runs an academic program during the summer (STARS) that complements the 
SEAS programs (which include the DC 21 st CCLC programs). The STARS programs run 
in the mornings, ending at 12:30 p.m., which is when the SEAS programs begin. Thus, 
students attending both programs are provided with full weekdays of activities during the 
six weeks of program operation. Unfortunately, three schools had their DC 21 st CCLC 
programs held in different locations than their STARS programs. 10 The decision to hold 
these programs in different locations was made by DCPS administration and was viewed 
as a challenge by the DCPS staff that managed the DC 21 st CCLC program. Parents of 
children whose programs were held in the same location expressed satisfaction at being 
able to drop off their children in the morning and know that they were supervised until 
6:30 in the evening. But in those several school programs where the programs were held 
in different locations, parents were inconvenienced. This problem may have affected the 
enrollment at the affected sites, according to APs. 

These three factors may have affected the program’s ability to serve its target population. 
The perseverance of the staff, in spite of these issues, suggests a healthy ability to flexibly 
implement the program and to deal with future challenges they are likely to face. 



9 Francis, Gamet-Patterson, and Terrell. 

10 The Sousa DC 21 st CCLC program was moved to Kramer, Francis was combined with that of Shaw, and 
MacFarland was scheduled to have its STARS and DC 21 st CCLC programs in different locations. 
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School/Community Issues 



Although most of our discussions with program staff concerned the summer component 
of the DC 21 st CCLC program, APs discussed two challenges to implementation of the 
after-school program: other neighborhood programs and safety. We discuss these issues 
here because they are relevant to the summer program as well. 

Other Neighborhood Programs 

The first issue concerns the effect of other programs, either within the host school or at 
other sites in the neighborhood, on the DC 21 st CCLC program. Three APs specifically 
mentioned that other programs “competed” with DC 21 st CCLC for students. One AP felt 
strongly that the District ought to “bring [all] community-level youth programming under 
the same umbrella” to reduce such competition. Another AP spoke to us about 
coordinating her summer program schedule and activities with those of surrounding 
neighborhood programs, indicating that it was challenging but not prohibitively so. In 
addition, some APs anticipated having difficulty recruiting the projected number of 
students because of the competing programs, a challenge discussed in our previous 
report. 11 

Safety 

One AP expressed continued concerns about the after-school program students going 
home after dark. This issue was also discussed in The Urban Institute’s July 2000 report 
on the after-school component of the DC 21 st CCLC program. This AP felt program 
enrollment suffers because parents do not want to have their children out when it is dark. 
She thought that the DC 21 st CCLC program may need to demonstrate a more realistic 
perspective about the neighborhoods it serves — although she did not make specific 
suggestions about how to do so. She also suspected that parents might not feel 
comfortable participating in the adult component (scheduled to start in Fall 2000), which 
is scheduled to run until 9:30 p.m. Finally, she suggested that the program may want to 
consider providing an on-site babysitter for the adult component to encourage parents to 
attend. 

Program Planning 

Although most of the APs and facilitators were satisfied that they had been able to 
implement their school programs as planned, they discussed ways to improve 
programming through more intensive planning at an earlier period. 



Some APs and facilitators felt that more planning would have improved the program. For 
example, two APs and two facilitators felt that they needed more time to plan the summer 
component together. One of these facilitators, who conducted athletic activities, said the 
program planning process needs to start three to four weeks in advance of the program’s 



11 Raphael & Chaplin (2000). 
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launch date. He also felt that the facilitators, at least in athletics, needed more than one 
planning meeting, particularly if they want to plan and offer “clinics” for students, with 
professionals who could share their expertise with students. 

Similarly, a third AP felt that her school staff needed additional general training, starting 
in advance of the program. Specifically, she felt that facilitators who are new to DCPS 
need more training. One “new” facilitator at another program agreed that he could have 
benefited from more training about program procedures, who to contact for problems, and 
other issues. Presumably these new facilitators were not able to attend the 4-day training 
provided by DCPS because they arrived too late. 12 

Overall, we had the impression that some staff (APs and facilitators) felt somewhat 
“rushed” to start up the summer component. Two APs said they needed more time to 
make the transition from the after-school to the summer component. One of these APs 
felt that the start date of June 26 th gave APs too little time (sixteen days) to wrap up the 
after-school program, and complete their proposals (including revisions requested by 
DCPS) in preparation for the summer program. Related to program planning, parents in 
our focus groups often suggested that the sites begin publicizing the summer program 
earlier in the school year. Some parents reported not learning of the program’s existence 
until after it had already started. On the other hand, starting the program later would 
mean that the children were left without access to this program for a longer period during 
the time when the regular school is not in session. 

In general, however, APs did not indicate that their programs were seriously 
compromised by these issues. 

DCPS Staffing Issues 

Several staffing issues related to the DCPS staff came up in our discussions with APs. 
These related to the number of staff at each site, the training and support they received, 
and the schedules for the APs. 

Number of Staff 

Nearly all APs and facilitators felt that they had hired a sufficient number of staff to serve 
the students in their program. However, two APs said that it would have helped them if 
DCPS staff had postponed making decisions about the number of facilitators to be hired 13 
until the second or third week of the summer program instead of during week one. At 
one site, many students delayed enrolling in the program after the first week, but the AP 
had already “cut” program staff to meet DCPS requirements for staff-to-student ratios. 
Staff had to be rehired, delaying program implementation somewhat. 



12 At least one facilitator specifically told us this. 

13 This decision is based on student enrollment. 



Staff Training and Support 

Most APs and facilitators were satisfied with the training and support they received to 
help them implement this program — although it was our impression that the training 
seemed somewhat unimportant to most of the staff we interviewed. Four of the nine APs 
reported that the professional development provided by DCPS helped them establish 
direction for their programs and serve as better program managers. The other five APs 
felt that the professional development had a minimal effect on their job performance, 
maintaining that they were “already” well qualified to do their jobs. One AP did suggest 
that DCPS staff consider having APs give more input on what type of professional 
development was offered to determine if these sessions could be more useful. 

In general, APs felt that they were given adequate opportunity to talk with DCPS staff 
about issues in the program. However, two APs reported tensions with DCPS staff. 

Most, though not all, facilitators in the summer program reported receiving two-hour or 
half-day orientations to the program from DCPS. Apparently they either arrived too late 
for the 4-day training or did not view it as “training.” In either case, facilitators had little 
to say about this training. One facilitator, however, said he would have liked a more 
thorough orientation to the program, particularly because he was not a teacher in DCPS. 
Athletic facilitators reported that the training they received from DCPS was useful. 

Schedule for APs 

Some APs expressed concern about how their days were scheduled. The APs were 
supposed to arrive at 12:30 p.m. each day, just before the children who would arrive 
between 12:30 and 1 :00 p.m. Two APs said that they did not feel comfortable arriving at 
the same time as their students because they needed to make sure that the site was well 
prepared each day before they arrived. They could have arrived earlier, but this would 
have been difficult given their late hours in the evening. After the program ends at 6:30 
p.m., APs were to participate in group meetings at Shaw Middle School, from 7 to 9 p.m., 
to discuss/review progress in the summer program and to plan the adult program, 
scheduled to begin in the fall. One AP suggested that DCPS try to avoid the evening 
meetings, as APs were too tired after working with their students earlier in the day and 
needed more time to work on preparing for and managing the summer program. 14 



School-Level Support 

We discussed issues related to facilities, equipment, program supplies, and staff support 
with the site-level staff. 

Facilities , Equipment , and Program Supplies 

Nearly all APs and facilitators reported that they had adequate facilities, equipment, and 
program supplies to implement their programs. There were two exceptions. First, two 

14 DCPS administrative staff pointed out that the APs were hired with the full knowledge that their 
schedules would involve the hours they served during the summer. 
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sites lacked DC 21 st CCLC computers. At one of these sites, students would walk across 
the street to the high school to use computers. At the other site, the procurement process 
was.delayed until the start of the school year and students did not participate in 
technology activities. Second, two APs reported that they were unable to write checks for 
supplies at the start of the program. In one case, this was a school-specific issue, and in 
the other, it was not clear why funds were not available to the AP. 

Staff Support at Site 

Nearly all facilitators reported that they received adequate support from their APs. They 
noted that their APs dealt quickly with requests for materials and facilities for activities, 
provided assistance with payroll issues, and “backed up” facilitators when disciplinary 
problems arose with students. One facilitator valued the fact that the AP voiced 
facilitators’ concerns to the DCPS staff who run the DC 21 st CCLC program. One 
facilitator did not feel well-supported by her AP but had not discussed her concerns with 
the AP because she started working in the program late and was not able to participate in 
the program planning process. 

Facilitators were assisted by aides in most activities. Some of these aides were 
Americorps volunteers, some were students (age 14-15) and a few were parents. Student 
aides were present but appeared to be underutilized in some activities. For example, we 
observed student aides sitting on the bleachers during aerobic movement/dance and art 
activities, and sitting alone in a classroom, with no students present, while the entire 
student body competed in a sports event in the gym. Although there may be limits to what 
these student aides can or should be doing in a program, it seemed to us that they could be 
trained to take on a greater role in the activities. However, many of the student aides 
either attended the same school or lived in the same neighborhood as the student 
participants. This familiarity may make it difficult to use them as supervisors or mentors. 
In addition, student aides were provided with an important job experience through DC 
21 st CCLC. 15 

The Americorps aides appeared to take a more active role. The technology activities at 
one site (the site that combined four school programs) utilized more aides, and in a 
greater instructional capacity, than other activities. Many of these aides were Americorps 
volunteers, which may account for the difference. We observed the Americorps 
volunteers providing a good deal of structured support to facilitators and students in the 
technology and art components. The Americorps volunteers were older and more 
experienced than the student aides, and their role in the program may have differed 
because of this. Facilitators occasionally relied heavily on the Americorps aides to teach 
and monitor the children. Indeed, DCPS staff told us that the Americorps volunteers 
were primarily responsible for the delivery of instruction in the technology components. 

In any case, all of the aides were highly valued by facilitators and APs. A technology 
facilitator mentioned that having aides was particularly important with the younger 



15 This was arranged in cooperation with the DC Department of Employment Services. 
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children, who need more help with computers than the older students. Several parents 
also spoke very highly of the Americorps volunteers in particular, saying that their 
children developed meaningful relationships with these individuals. 



IV. Observed Features of Student Activities 

O ur site visits included observations of 30- to 60-minute segments of 22 student 
activities conducted as part of the DC 21 st CCLC program (at least two activities 
at each school program). Appendix A provides the dates, times, staffing, and number of 
students in these observed activities. 

We begin by providing some common features we observed in student activities at the 
sites, followed by a discussion of the structure of the activities. 

Common Features of Activities 



■ Observed student activities were implemented with sufficient staff supervision and in 
adequate spaces, with sufficient time and equipment. Most of the activities we 
observed involved small numbers of students, typically 12, with a range of 4 to 20, 
with an average of around 13 (not including gym activities). Most activities were led 
by one facilitator who was assisted by some number of aides (usually between one 
and four), depending on the type of activity and the size of the group. Most of these 
activities were conducted in a sufficiently sized space such as a classroom or in an 
area of an “open air” school. Materials and equipment were adequate. The computer 
facilities, in particular, were quite sophisticated at some sites, with computer desks 
that used a glass screen and special housing to protect the computers from damage. 
Activity periods generally were one to one-and-a-half hours, with one site offering 
activities for three-hour periods of time. 

■ Activities were well planned. Even when activities involved “free” time for students 
to use the Internet or play sports, the students had a routine to follow and did not stay 
idle for substantial amounts of time. There were exceptions, of course, but these were 
few. 

■ Nearly all of the activities involved hands-on “work” rather than listening to an 
instructor or passively watching a presentation. There were exceptions, but in general 
students were doing various things, including dancing, singing, doing artwork, using 
computers, and playing educational games. They frequently took responsibility for 
leading one another in a task, such as leading peers in a series of athletic stretching 
exercises. 

■ In general, facilitators and aides maintained discipline and kept students on task. 
However, as expected in any program, some facilitators exercised better “classroom” 
management than others. In two activities we observed what appeared to be poor 
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classroom management. Both of these activities involved the older students in junior 
high/middle school, suggesting that what appeared to be poor classroom management 
may have been caused, at least in part, by difficulties related to managing older 
students. 



Structure of Student Activity Groups 

In most cases, the groups in which students participated in activities were of two types, 
either small or large. Most groups were small, with students of a specific age group, and 
were monitored closely by one facilitator and several aides. As mentioned, “class” sizes 
(except in gym) ranged from 4 to 20 students. In these small groups, students did creative, 
athletic, and/or educational activities, such as weightlifting, dancing, creating artwork, 
playing educational games, using technology, etc. Students worked independently in 
many activities, including technology activities, arts and crafts, and weightlifting. In other 
small “classes,” students worked in even smaller groups playing educational games, 
competing or playing in sports, singing, and dancing. In several of the observed activities, 
students were rehearsing performances (such as dances and fashion shows) that would be 
performed for other students and their families at the end of the summer. 

In contrast, other groups — particularly for athletic events — included larger numbers of 
students and received less direct supervision by facilitators and/or aides. In some athletic 
events held in school gymnasiums, for example, students were given access to a variety of 
sports equipment (such as jump ropes, basketballs, and lacrosse and boxing equipment) 
and were permitted to use this equipment appropriately on their own. For example, we 
observed over 25 students in one activity at one site and about 60 at another. In both 
cases the students were of various ages and were playing different sports. They obtained 
the equipment themselves and used it while being supervised by facilitators. However, 
the facilitators did not monitor the students so much as maintain order. In other athletic 
activities, students in large groups competed against one another, as in a large “horseshoe 
tag” game involving approximately 40 students, ages 8-13. 16 In this case, one facilitator 
“presided” over the game, providing encouragement and promoting good sportsmanship 
and teamwork through verbal feedback to teams and individual students. 

Although students were typically grouped in these two ways, several features of service 
delivery enhanced the potential for learning and interacting. These features include the 
following: 

1 . In many activities of both types, students were given choices about what they would 
do during the activity period. For example, students in one activity could play chess or 
checkers, use the computer to play educational games (using the Leamstar program, 
in this case), or sit quietly and color. Students were able to start with and switch to 
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Another 40 were watching. 



whatever specific activity they wished. Significantly, we observed them persisting in 
each activity rather than switching frequently. 

2. In a significant number of activities, particularly technology, students were permitted 
to work individually and test their own limits in^various tasks. For example, 
facilitators assigned students a particular lesson from the Almena typing tutorial but 
did not appear to monitor students’ progress continually. Occasionally, students were 
asked to take a timed typing test that gave them feedback on their progress. In the 
weightlifting activity, the facilitator emphasized that students should select the 
appropriate weights for each exercise, based on their individual capacity. 

3. Students were also encouraged to compete with each other in a number of areas. 
Sports competitions were organized by Mr. Briscoe, as discussed earlier. Students 
played competitive games, such as chess, checkers, and tic-tac-toe. In addition, 
results from the Almena tests and from the stock-market game were stored 
electronically and students were told how they were doing both absolutely and 
relative to each other. 

4. Some activities were structured to satisfy the special interests/needs of specific groups 
of students. For example, a weightlifting activity was geared toward a group of male 
students who wished to participate in the school-year football team. Their workout 
routine was more rigorous than that of other coed groups, according to the facilitator. 
In another example, students with a strong interest in participating in a fashion show 
were given this option, whereas other students did not participate in this activity. 
Students who were particularly interested in mural-making were observed being 
pulled out of other activities to complete a wall mural at one of the school sites. 

5. Often, students were permitted to interact freely with their peers. Many activities were 
structured to give students independent learning opportunities but also the freedom to 
interact freely with one another. For example, in one art activity, 7- and 8-year-old 
students freely observed one another’s projects and gave feedback to each other. 

Perhaps more significantly, students typically interacted with their preferred peers, 
their friends, something that may be less prevalent in a regular school setting. For 
example, in one educational activity, fifth-graders played several games involving 
mathematics. Three to four students sat at tables in groups of their own choosing, 
often in single-sex groups. We were impressed with how well-behaved and engaged 
the students remained throughout the period. 

Permitting students to interact with their friends may also give them an opportunity to 
practice important social skills while learning. In the educational activity mentioned 
above, a conflict arose between the four boys seated together. One boy, “A,” 
announced that one of his opponents had stolen one of his game cards. For about five 
minutes, the students discussed this problem, making several useful suggestions as to 
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how to resolve the problem. The facilitator came over to their table and helped them 
reach a solution together. As she walked away, the boys continued playing. 

This example demonstrates the value of the facilitator’s presence to these students — 
the “discussion” may well have degenerated into a fight if no adult had intervened. 
Yet in this setting the students were able to exercise conflict resolution skills on their 
own for some time before the facilitator intervened. In a more regimented setting, 
where students are told to work in silence or are not allowed to work with friends, 
such an opportunity may not have arisen. 

In another example, a facilitator observed young boys refuse to be dancing partners 
with young girls. The facilitator made some creative suggestions, including some 
“rules” for the dance, which led to some cross-gender dancing. 



V. Potential Program Outcomes 

O ne of the purposes of our visits to the school sites was to gather information 
about possible outcomes of the program, particularly for students. The 
information supplied here is not designed to show program effects. Instead, we provide 
this information to stimulate further investigation. The information is based on our 
observations and comments made by parents and/or guardians and facilitators about the 
program. A rigorous evaluation would attempt to demonstrate whether these outcomes 
differed between students with access to the program and those in an appropriate 
comparison group. 

Short-Term Outcomes 

These are outcomes that we believe could be observed as immediate benefits of the 
program. Many of these may also affect student outcomes in the longer term. For 
instance, increased engagement may improve student learning. 

Student interest and engagement in activities. Although students were clearly more 
enthusiastic about some activities than others, they appeared to enjoy nearly all of the 
activities we observed. Students demonstrated high levels of engagement in these 
activities and were spending most of their activity time discussing the tasks at hand with 
one another. Parents and/or guardians also provided numerous examples of their 
children’s positive reactions to the program, including being prepared to go early, telling 
their parents not to pick them up before the program ended, and discussing specific 
program activities at home with their parents. Some parents expressed surprise that their 
children had remained interested in the program throughout its duration, indicating that 
other summer programs had not held their children’s interest for so long. 
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Based on the reports from parents in the focus groups, it appears that students would 
demonstrate high participant satisfaction using surveys or interviews (as was found in 
focus groups with students in the after-school site visits). 

Providing free, high-quality care for school-aged children. Many parents and/or 
guardians indicated that by providing free child care, the summer program had some 
benefits for families. Some parents said they would have had to pay (e.g., $50 per week) 
to enroll their children in other summer programs. Parents also appreciated having some 
time away from their children, which reduced stress in their homes. In some cases 
children from the same family were able to be together in the program, which parents 
viewed as convenient and beneficial for the family. 

Students 9 comfort level with technology. Students appeared quite comfortable using 
computer technology in the activities we observed. They started working on their 
assignments quickly, with the facilitator helping them with equipment only if needed 
(e.g., setting up a headset for the typing CD). Clearly, a routine had been established by 
the facilitators, and students followed this routine well. 



Longer-Term Outcomes 

Longer-term outcomes are those that may benefit students in the long run, though many 
may also have short-run benefits. For instance, learning about new activities (the first 
potential outcome) can be fun and can also open up more options for future activities for 
these youth to participate in. 

Learning about new activities. Although we did not survey students about their previous 
experiences, we are fairly certain that many had not been previously exposed to many of 
the activities available in the DC 21 st CCLC program, such as lacrosse, ballet, and various 
forms of “community service.” (Students in focus groups in the after-school program 
frequently indicated that program activities were new to them.) Student participants were 
given the opportunity to try new activities and to learn about them. For example, one 
athletics facilitator said he doubted many of his students had access to weight machines. 
He hoped that by emphasizing the concepts underlying weight training, he had helped his 
students understand the health benefits of weightlifting. 

Developing collaborative and cooperative skills. We were impressed with the high level 
of collaboration that occurred in many student activities, particularly among the older 
students, such as in developing an original dance to be performed for students and their 
families. In addition to collaboration, students frequently supported one another, as when 
young children helped each other clap out a rhythm in a music activity or helped one 
another use the computers. In technology activities, we observed a good deal of 
spontaneous helpful behavior between students. For example, during free time devoted to 
“surfing the Net,” students shared information and tips about how to navigate the Web. 
And as discussed in the previous section, students were observed exercising conflict 
resolution skills while playing games with one another. 
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Learning social skills. Parents and/or guardians were pleased with the potential for good 
socialization that is part of the program. They appreciated that their children were 
spending time with students from different schools and of different ages (students of 
different ages are combined during field trips, snacks, and other large-group activities) 
and from different schools. Some said their children were around a “better group of 
students”, and had the opportunity to establish “more positive” friendships in the 
program. Some parents felt their children might not engage with other children if not 
enrolled in a program such as DC 21 st CCLC. 

The program also provided a great deal of time for the children to interact with adults in a 
low-pressure environment. In nearly all activities in the smaller groups — and frequently 
in the larger groups — students received individual attention from adults. We observed 
facilitators providing “instruction” in every activity, as well as encouragement, advice, 
and constructive feedback. Several parents felt that the Americorps volunteers were good 
role models for their children. 

In addition to teaching them to get along with others, the DC 21 st CCLC program 
provided an opportunity for the youth to work to help others. The community 
involvement component of the DC 21 st CCLC program encouraged the youth to do 
community-oriented activities such as cleaning up the school area and visiting homes of 
the elderly. Such activities may increase the chances that these youth continue to work to 
develop their communities in the long run, thereby benefiting both themselves (because 
of the opportunity to socialize) and those with whom they share these communities. 

Increasing self-confidence. Some parents reported that during their participation in the 
program, their children had begun displaying increased confidence at home. They were 
“coming out of their shells,” by dancing, singing, and expressing more of their opinions at 
home. These parents could not be certain that these changes would not have occurred in 
- the absence of the program, however. 

Increasing ability to focus. Several parents felt that their children were learning to be 
calmer and possibly more focused by pursuing enjoyable activities while in the program. 
These parents had children with attention-deficit disorders and/or behavioral difficulties. 

Developing academic skills . In technology and other activities, we observed students 
engaged in the development of basic academic skills, including reading, writing, using 
mathematics, and problem-solving. While these are primarily the focus of the schools, 
approximately one-fourth of DC 21 st CCLC program time is devoted to development of 
academic skills. A particular strength of the DC 21 st CCLC program could be the high 
level of use of up-to-date technology. Specific skills covered included becoming 
computer literate, typing, using the Internet to search for information, purchasing stocks, 17 



17 The children did not purchase real stocks. Rather, they played a computer game that simulated stock 
market transactions using the true market prices. 
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and using e-mail. More detailed observations of this feature of the program are provided 
in Appendix B. 



VI. Conclusion 

X he DC 21 st CCLC program has continued its progress toward full 

implementation in the District of Columbia during the summer of 2000. 
Activities were provided for about 1,000 children attending nine middle and 
junior high schools and their feeder elementary schools in the areas of education (reading, 
math, and technofluency), sports, arts, and community service. These activities built on 
past after-school and summer school programs by helping to develop a “seamless web” of 
activities. When fully implemented, this web should help to ensure that students are not 
left wanting for productive things to do during the summertime when their schools close, 
and when many youth drop behind academically. 

The success in implementing the program was accomplished in spite of a number of 
significant challenges. In particular, a number of schools were closed for the summer and 
the other major summer program for youth (the STARS program) could not be co-located 
with the DC 21 st CCLC programs in three schools. It is very likely that the enrollment 
figures would have been higher in the absence of these challenges. 

Planning, staff support, and staff development all appeared to be proceeding smoothly. 

We observed evidence of adequate preparation and support, and staff reported being 
generally satisfied, though some staff requested more staff development, or a different 
schedule for staff development, suggesting that there may still be room for improvement. 

Parent reviews of the program (as provided in focus groups) were enthusiastic. They 
were particularly thankful for the fact that the program was provided at no cost and was 
providing their children with structured activities. Many parents reported observing 
positive changes in their children’s behavior that they attributed to the program. 

Our site visits revealed that the student activities held promise for producing positive 
outcomes in students. We observed students engaged in a wide range of learning 
opportunities and interacting socially in a low-stress environment that was free from 
disruptive conflict. In addition, the staff generally appeared active and well-utilized. A 
particularly striking feature of the program that we noted was the use of technology. 
Students at all but one site had access to computers where they could develop their 
general computer, typing, Internet, and e-mail skills, all of which are crucial for most 
good jobs given the current economic environment. 

Of course, we cannot describe what sorts of experiences these youth would have had in 
the absence of the program since we had no data on comparison sites. This points to the 
importance of a carefully designed impact evaluation, which is currently being designed. 

In addition, as stated in our earlier report on the after-school component of the DC 21 st 
CCLC program, continued self-monitoring can help provide useful information to the 
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DCPS staff who run the program, to the rest of the DC Children and Youth Investment 
Partnership, and to individuals involved in helping to design and manage similar 
programs in other cities. 
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Appendix A: Characteristics of Observed Program Activities 
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Appendix B 

Technology Use in DC 21 st CCLC Program 

Since the DCPS staff have a strong interest in developing the technology component of 
the program, we observed as many technology activities as possible — in total, 30 to 60 
minutes of six technology activities involving students and staff from six different school 
programs. In most of these activities, students used one or more of the following 
programs: 

1) A self-guided typing tutorial (Almena CD). 

2) The Autoskills and Leamstar programs, which offer games with academic skill- 
building content. 

3) The Youthline USA Web site, where they read e-mail, visited Web sites of their own 
choosing, accessed educational articles and information, and used a simulated stock 
exchange for which students maintained their individual stock portfolios. 

We observed most students undertaking these activities in a focused manner. Some of the 
older students were more likely to “surf the Net” to find and play music — an activity that 
is encouraged by facilitators and aides. However, one technology facilitator said she was 
frustrated sometimes with the older students. She tried to emphasize to these students that 
they must do the assigned activity before they can have “free” time. She also indicated 
that most of her students, of all ages, were not motivated to learn to touch type. 

Given the relatively small number of observations we conducted, we do not know what 
kind of follow-up or reinforcement facilitators provide for the students. One facilitator 
shared with us the reports his students created that combined research on the Internet, 
typing skills, and reading comprehension skills. But we were not present at the sites long 
enough to examine many examples of extended learning projects. 

At the same time, we raise the following question: How can the facilitators determine 
whether students are learning new technology skills? In several activities that we , 
observed, students were looking at their hands while they typed and/or were not using the 
“home” keys. Thus, technically, they were not practicing touch typing. (This of course 
does not mean that their time was wasted. Learning to touch type may be a gradual 
process.) Another question we raised after conducting our observations was whether the 
typing tests, as they are implemented in the program, affect students’ learning positively 
or negatively. One facilitator reported that students aren’t motivated to learn touch typing 
(she was referring to a group of 12- to 13-year-olds). Another facilitator felt that some 
students became unruly in class because they were nervous about not being able to pass 
the typing test. 

In another activity, students who had been told to “surf’ the Net spent a good deal of their 
time listening to their favorite music. Will this help the students develop their technology 
skills? In one activity it was not clear whether students were reading the educational 
article that the facilitator asked them to access on a Web site. (We were not present the 
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next day to see if the facilitator discussed the article with them.) Furthermore, during 
“free” time, students were able to download and listen to music, including music with 
profane lyrics, without adult censorship. Music with inappropriate lyrics was downloaded 
from Web sites such as “Hitlist,” which we were told is not blocked by the DCPS 
system’s filtering mechanism. 

To summarize, the DC 21 st CCLC program has implemented a strong technology 
component that utilizes a wide range of software. This program was well in place at most 
of the sites and appeared to be generally well-managed. 
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Appendix C 
Protocols for Site Visits 
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Assistant Principal Interview Protocol 



Introduction 

This interview is similar to the one we conducted with you in May. The purpose of our interview 
with you today, which should last about 20 minutes, is to understand the ways in which the 
summer program is and isn’t working as well as anticipated, and any lessons you’ve learned. 



Do you have any questions for us before we begin? 

General Questions 

We’re going to start with some general questions about the DC 21 st CCLC program. 

L What types of professional development activities were you given to prepare for 

the DC 21 st CCLC program (summer or school year) ? 



II. Do you feel that these professional development activities have affected how you 

do your job? If so, how? 



Summer Program Questions 

The rest of our questions refer only to the summer program. 

III. Has the program ’s Neighborhood Advisory Council been formed and is it 
functioning? 

If yes, what role are they playing in the program? 

IV. How were students recruited to participate? 

Probes: Was there a target group? Did you send flyers home with students during school year? Were 
parents contacted? 



V. Were more students interested in participating in the program than you have 
been able to serve? Please explain. 

VI. How were the facilitators recruited and selected? 
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VII. Are any of the student activities organized to encourage competition? How? 



VIIL What data are being collected on participants, formally or informally, such as 
student attendance records, a behavior log (find out positive/negative)? 

IX, Which aspects of the program have been most difficult to implement? Can you 
give examples? 

Probes: What obstacles, challenges, or conflicts have you faced? 

Facilities, staff quality and skills, space 

X, Were these challenges overcome? If yes, how? If no, why not, do you feel? 



XI, Do you have any suggestions for ways to improve the program ? Please explain. 



Thank you for your time. 
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School: 

Date: 

Parent Focus Group Protocol 

Introduction to parents to explain purpose of interview , describe confidentiality, and ask 
permission to tape discussion. 

Introductions: 

1. Please tell us who you are and a little about your child (or dependent) who is participating in the 
program (including age, gender, school, extent of participation in program). 

Reasons for Participation: 

2. Did your child want to participate in the summer program? Why or why not? 

3. Did anyone else (including you) suggest that your child participate? If so, why? 



Experiences in the Program: 

4. Please tell us about your child’s participation in the program. What do you know about it? 

Probes: What does your child do? What does your child tell you about the program? 

5. How excited is your child about attending the program? Can you give examples? 

Benefits: 

6. What benefits, if any, has the program had for your child? Please give examples. 

7. [If parents had specific reasons for their children participating (Q3), read them back.] Are your 
children gaining what you had hoped they would from the program? 

8. Are there any attitudes or behaviors of your child that you feel have changed, or could change in 
the future, as a result of his/her participation in this program? 

For example, attitudes might include your child’s interest in learning math, using computers, or 
talking to community leaders. Behaviors might include the amount of time your child spends 
watching television, reading, talking on the phone, fighting with other students, or trying to resolve 
conflicts peacefully. 

9. Have any of your child’s attitudes or behaviors actually changed? Can you give examples? 

Suggestions: 

10. Do you have any suggestions about how the program could be improved? 

Probes: Ways to get more students involved? Ways to help the students more? Ways to get more 
parents involved? 

Data Gathering: 

11. As part of our ongoing efforts to improve this program, we hope to collect more information in 
the future about the program from parents. How would you suggest we collect such information? 

Probes: Through surveys? If so, by mail to homes? Through telephone interviews? Other means? 

12. We are thinking about sending out surveys to children and parents. What kind of information do 
you think we can and should collect on such surveys? 



Thank you. 
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School: 

Activity/Facilitator: 

Date: 

Observation Guide Protocol 



Time activity starts: 

Duration of activity.* 

Time observation starts: 

Time observation ends: 

Number of boys: 

Number of girls: 

Age/grade of students: 

Number of facilitators: 

Number of aides: Adult H.S. student Male Female _ 

Initial Information : 

1. Describe the setting (room, building, field, non-school location etc.) for the 
activity. 



2. If you were present at the beginning of the activity, how were expectations for the 
activity communicated by the facilitator (if at all)? What did you think was the 
“plan” for the activity period? 



3. Did students seem to know what was expected of them (e.g., did they quickly 
engage in the activity, ask questions, appear interested)? 
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Monitoring of Activity : (To be noted every 10 minutes during the course of the 
observation.) 



Please keep a running log of what is occurring in the activity, using the questions 
below as a guide. Also make use of the features described in the “Plugging In” 
report and consider the summary section that follows. Please answer the questions 
directly every 1 0 minutes. 

4. Describe the activity , providing details about what materials are used, what 
manipulatives, what the actual activity is, how the activity is being accomplished, 
etc. 

5. What are the students doing (e.g., acquiring information, practicing for a 
performance, discussing a community service project)? 

6. How are students organized (whole-group, small groups, pairs, working 
independently)? 

7. How are students interacting with one another ? Be sure to note examples of 
collaboration, cooperation, praise, feedback, shared responsibility for activity. 



8. How is the facilitator interacting with students ? How are aides interacting with 
students ? Be sure to note facilitation, feedback, co-learning, guiding. 



9. Approximately how many students are engaged in the activity at hand? What are 
the unengaged students doing? 

[Note: This page is repeated 5 times.] 
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Summary : 

10. Describe the noise level. Does it appear to be a problem or not? 

11. How is time spent during the observed period? 

Percent of time spent doing hands-on activity: 

Percent of time spent doing physical activities ( 

Percent of time spent on competitive activities ( 

Percent of time spent in group work among students: 

Percent of time whole group spends listening to facilitator 
Other ( ): 

12. Do students have the opportunity to take responsibility for any aspects of the 
activity? Do they do so? 

13. What did you perceive as benefits of this activity for students? Please explain, 
using examples, quotes, description, etc. 

14. How interested are students in doing this activity? Please provide examples and 
support for your assessment. 

15. Would you describe this activity as structured, semi-structured, or 
unstructured? Why ? 

16. How engaged are students in this activity overall? 

Percent of students very engaged: 

Percent of students somewhat engaged: 

Percent of students not engaged: 

17. For students somewhat or not engaged, what are they doing (e.g., talking to each 
other, daydreaming, walking around room)? 

18. Does the facilitator make an effort to re-engage students who are not engaged? 
Do other students? How? 

19. How would you describe the degree of supervision received (e.g., from students 
actively supervised to neglected)? 

20. How is discipline maintained? How effective was the facilitator at managing the 
students? Please support your assessment. 

21. Were any students told to leave the room/area during this activity because of a 
discipline problem? Explain. 

22. Is the space adequate, in your opinion, for this activity? If not, why not? 

23. Are there adequate materials for all students (e.g., sports equipment, art 
supplies, space, etc.)? Explain. 
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24. Was the community involved in any way with this activity (e.g., students 
preparing artwork for a retirement home)? Explain. 

25. Describe any interesting/relevant occurrences during this activity: 

26. Record any particularly interesting/relevant statements made hy students or 
staff: 

27. If food was served, please answer the following: 

What food was served? How much? 

Did it appear fresh? 

Did it appear healthy? 

28. If parents were involved, please answer the following: 

How many parents were involved? 

What were the parents doing? 

Did the parents seem interested in the activity? 

Were they interacting with their own children? 

Were they interacting with the children of other parents? 



Thank the facilitator. 
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